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C2  of  Unmanned  Systems  in  Distributed  ISR  Operations 


Abstract 

This  paper  describes  a  series  of  experiments  to  investigate  issues  of  human-robot  teaming  and 
network  centric  operations.  Experiment  objectives  were  coordinated  to  address  issues  within  and 
among  the  physical,  communications,  infonnation,  and  human  (cognitive/social)  domain  layers 
of  the  network.  Objectives  spanned  the  cognitive,  social,  and  physical  domains  of  the  network. 
In  the  cognitive  domain,  researchers  tested  a  predictive  performance  tool  for  robotic  operators 
and  measured  operator  situational  awareness  and  workload  during  missions  as  these  conditions 
related  to  reliance  upon  unmanned  surveillance  technologies.  In  the  social  domain,  we 
documented  the  ad  hoc  development  of  social,  task,  and  knowledge  networks  during  missions. 
These  human  dimensions  of  the  network  were  juxtaposed  to  the  agile  computing  infrastructure 
operating  over  a  Future  Force  surrogate  network  and  an  802.11  network.  Results  show  that 
many  challenges  exist  across  the  layers  of  the  network  domain  architecture.  Primary  among 
these  is  to  develop  a  mobile  ad  hoc  network  (MANET)  to  support  mobile  and  extended 
vehicle/dismount  ranges  in  a  variety  of  terrain  conditions.  In  the  cognitive/social  domain,  we 
need  to  understand  what  infonnation  Soldiers  need  from  a  network,  when  this  information  is  of 
maximum  use,  and  what  fonn  the  information  should  take  for  maximum  situational  awareness 
and  decision  making. 


Introduction 

This  report  details  the  human  factors  analysis  conducted  in  July  2007  at  an  experiment  designed 
to  investigate  the  impact  of  a  suite  of  unmanned  C4ISR  technologies  on  platoon  force 
effectiveness,  as  measured  by  individual  understanding  of  networked  information.  The 
unmanned  technologies  included  in  the  experiment  architecture  were  the  PackBot  Small 
Unmanned  Ground  Vehicle  (SUGV)  and  a  Family  of  Unattended  Ground  Sensors  (FUGS).  The 
objective  of  the  human  factors  assessment  was  to  evaluate  the  contributions  that  unmanned 
technologies  make  to  situational  awareness  when  used  in  a  relevant  field  environment  and 
investigate  the  cognitive  abilities  that  appear  to  be  correlated  with  high  perfonnance  on  tasks 
associated  with  tele-operating  a  SUGV  under  non  line  of  sight  conditions.  This  contribution  to 
understanding  the  human  dimension  of  network  centric  operations  is  critical  to  establish 
communications  capabilities  to  connect  decision  makers  with  essential  elements  of  information. 


Method 

Situation  Awareness  and  Workload  Measures 

In  examining  how  unmanned  systems  impact  platoon  effectiveness  with  respect  to  Intelligence, 
Surveillance,  and  Reconnaissance  (ISR)  missions,  ARE  researchers  and  technology  developers 
were  specifically  interested  in  understanding  how  a  suite  of  unmanned  technologies  contribute  to 
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Soldiers’  ability  to  develop  situational  awareness  (SA).  Although  SA  is  difficult  to  measure  in  a 
field  setting  because  of  the  need  to  divert  Soldiers’  attention  away  from  battlefield  activities 
(Bowman  &  Kirin,  2006),  a  newly  developed  survey  methodology  utilizing  standard  Army  ISR 
reporting  techniques  was  implemented.  This  survey  methodology  uses  the  military  standard 
SALUTE  format  with  added  questions  to  probe  Soldiers’  assessment  and  prediction  of  known 
battlespace  activities.  1  These  surveys  (titled  the  SALUTE-AP;  Bowman  &  Thomas,  2006)  were 
given  to  each  Soldier.  Each  survey  was  administered  at  approximately  20-minute  intervals  that 
coincided  with  natural  or  planned  pauses  in  the  scripted  OPFOR  activity.  One  analyst  was 
assigned  to  the  OPFOR  and  recorded  ground  truth  as  the  mission  progressed.  This  ground  truth 
survey  was  used  to  score  each  individual  Soldier  SALUTE-AP  at  the  conclusion  of  the  mission. 
The  survey  was  scored  on  a  3  point  scale  (l=low  SA,  3=high  SA),  and  provided  an  objective 
measure  of  Soldier  SA. 

SA  of  platoon  leaders  has  been  successfully  measured  in  the  field  using  the  Mission  Awareness 
Rating  Scale  (MARS)  (Matthews  &  Beal,  2002).  The  MARS  instrument  includes  two  subscales 
that  assess  SA  content  and  SA  workload.  Each  subscale  includes  four  questions  that  address  the 
three  levels  of  SA  (identification,  comprehension,  and  prediction).  The  fourth  question  is 
concerned  with  how  well  mission  goals  were  identified.  The  four  workload  subscale  questions 
are  aimed  at  asking  respondents  to  quantify  the  effort  involved  in  making  decisions  and 
predictions  regarding  their  mission  as  well  as  identifying  and  comprehending  the  infonnation  as 
presented  to  them.  Matthews  and  Beal  reported  that  the  MARS  was  effective  in  differentiating 
between  squad  and  platoon  leaders’  SA  in  a  field  training  environment.  The  measure  was  also 
reported  to  be  unobtrusive  and  easy  to  administer,  requiring  less  than  5  minutes  to  complete 
(Matthews  &  Beal,  p.  8).  In  this  experiment,  we  did  not  administer  the  workload  subscale  and 
instead  used  the  NASA  Task  Load  Index  to  measure  perceptions  of  workload.  This  measure 
requires  participants  to  rate  their  workload  according  to  six  dimensions.  These  are  mental  and 
physical  workload,  time  pressure  felt,  satisfaction  with  own  performance,  effort,  and  frustration 
felt.  These  questions  are  measured  on  a  10  point  scale.  The  participants  completed  the  MARS 
and  the  NASA  TLX  at  the  conclusion  of  each  mission,  or  once  per  day. 

Training 

Soldiers  received  one  week  of  hands-on  training  prior  to  the  start  of  the  test  runs.  This 
training  included  hands-on  instruction  on  multiple  unmanned  technologies  [two  PackBot 
SUGVs,  and  six  FUGS].  A  refresher  course  was  given  on  the  Force  XXI  Bahle  Command 
Brigade  and  Below  (FBCB2)  version  4.4.2  battle  command  system.  Each  soldier  received 
specialized  training  in  operating  the  PackBot  SUGVs.2  Soldiers  were  given  a  demonstration 
about  the  PackBot’ s  capabilities;  its  intended  uses  in  the  field;  and  the  types  of  operator  or 
control  problems  to  anticipate  as  well  as  the  appropriate  techniques  for  problem  solving.  This 
training  session  also  included  details  about  the  Operator  Control  Unit  (OCU).  After  the  one  hour 


1  The  SALUTE  report  is  an  acronym  for  the  traditional  categories  of  a  tactical  report:  Size,  Activity,  Location, 
Uniform,  Time,  and  Equipment.  We  added  the  categories  of  Assessment  and  Prediction.  For  each  of  those  two 
latter  categories,  we  asked  Soldiers  to  express  their  level  of  confidence  in  their  answers,  measured  on  a  5  point  scale 
(1  =  low,  5  =  high). 

2  Each  Soldier  was  trained  in  the  SUGV  in  order  to  participate  in  the  cognitive  test  battery.  Only  two  of  these 
Soldiers  used  the  SUGV  in  actual  mission  trials  in  support  of  the  larger  activity. 
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lecture  style  training  sessions,  each  Soldier  was  instructed  to  drive  the  PackBot  line  of  sight  and 
non-line  of  sight  inside.  The  one-on-one  driving  instruction  included  a  series  of  performance 
tasks.  The  primary  task  was  to  drive  the  PackBot  up  and  down  a  short  flight  of  stairs,  about  one 
foot  high.  Soldiers  became  familiar  with  raising  and  rotating  the  head  of  the  PackBot  as  well  as 
the  type  of  system  feedback  provided  on  the  OCU. 

Additionally,  two  Soldiers  were  trained  in  operating  the  FUGS.  These  Soldiers  were 
selected  by  their  platoon  leader  because  they  had  experience  with  similar  sensors.  At  the 
conclusion  of  the  training,  the  technology  developers  verified  that  the  Soldiers  assigned  to 
operate  the  technologies  were  sufficiently  trained  to  safely  operate  the  vehicles. 

Also  during  this  week,  Soldiers  were  given  four  blocks  of  training  on  the  Army  Cognitive 
Readiness  Assessment  (ACRA)  system.  The  objective  in  this  phase  of  training  was  to  allow 
each  Soldier  to  become  familiar  with  how  to  take  four  cognitive  tests  via  a  lap-top  computer 
(O’Donnell,  Moise,  &  Schmidt,  (2004)).  Each  block  of  training  lasted  15  minutes.  Soldiers 
were  given  one  (1)  30  minute  break  in  the  middle  of  training,  resulting  in  a  total  training  time  of 
no  more  than  90  minutes.  At  the  conclusion  of  the  week,  each  technology  expert  verified  that 
the  Soldiers  are  sufficiently  capable  of  operating  the  system  with  minimal  support  and  the  ARL 
scientist  verified  that  the  Soldiers  understood  how  to  complete  each  test  in  ACRA. 

Scenarios 

Missions  were  focused  in  the  ISR  domain.  Five  scenarios  were  developed  to  address  the 
objectives  described  above;  each  relied  upon  the  presence  of  an  active  but  scripted  Opposing 
Force  (OPFOR).  Three  types  of  test  ranges  were  employed,  spanning  the  range  of  possible 
environments  at  the  test  facility.  These  are  described  in  Table  1.  Each  mission  was  conducted  in 
the  same  manner.  Soldiers  would  load  into  vehicles  at  the  assembly  point  and  convoy  to  the 
designated  test  site  for  that  day.  They  would  unload  unmanned  systems  and  arrange  vehicles  as 
designated  by  the  test  director  to  maximize  network  connections.  When  the  Soldiers  were  in 
position,  the  OPFOR  was  directed  to  enter  the  test  range  and  conduct  activities  as  directed  by  the 
test  director.  Surveys  were  administered  during  and  after  the  mission  as  detailed  below.  The  test 
director  called  an  end  to  the  mission  when  all  activities  were  concluded. 

Table  1  Test  Locations 


Test  Range 

Characteristics 

Dates 

Case  1,  TAC  9D 

Open  rolling  sandy  areas  with 
lightly  forested  sections. 

18  and  25  July 

Case  2,  TAC  12A 

Open  sandy  areas  with  heavily 
forested  sections  including  ‘Vietnam 
Village’,  a  set  of  three  huts  in  a 
forested  area. 

19  and  20  July 

Case  3,  MOUT 

An  urban  site  configured  in  a 
triangular  shape. 

24  July 
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Demographics 


10  Soldiers  from  the  New  Jersey  Army  National  Guard  participated  in  this  experiment. 
These  Soldiers  were  members  of  one  platoon  and  therefore  knew  each  other  well.  Their  average 
age  was  29  (SD  =  9.3).  This  was  an  Infantry  company;  all  Soldiers  were  male.  All  but  one 
Soldier  reported  a  military  occupational  specialty  (MOS)  of  1  IB.  More  than  half  (N=6,  60%)  of 
the  Soldiers  reported  less  than  five  years  of  military  service.  Three  Soldiers  reported  between  5 
and  15  years  of  military  service  and  one  reported  16  or  more  years  of  military  service.  Sixty 
percent  of  the  Soldiers  (N=6,  60%)  were  NCOs  (E5  and  E6).  Four  Soldiers  were  lower  enlisted 
ranks,  one  E3  and  three  E4s.  Four  (40%)  of  the  Soldiers  reported  having  a  high  school  diploma. 
Three  had  an  undergraduate  college  degree,  one  had  taken  some  graduate  courses  and  two  had  a 
graduate  degree.  These  Soldiers  reported  that  they  had  little  experience  with  unmanned  systems 
and  a  communications  network  except  for  their  participation  in  PM  C4ISR  OTM’s  2006  activity, 
in  which  most  of  these  Soldiers  took  part. 

Army  Cognitive  Readiness  Test  Experiment 

During  the  first  week  of  the  training  phase  of  the  experiment,  a  small  excursion 
experiment  was  conducted  to  investigate  if  scores  from  a  cognitive  test  battery  could  be 
correlated  with  performance  operating  a  SUGV.  Literature  in  human  robotic  interaction 
consistently  show  that  individual  cognitive  attributes  such  as  spatial  orientation  and  visualization 
(Gugerty,  2004;  Chen  et  ah,  2006),  time  velocity  estimation  (Rastogi,  1996;  Darken,  Kempster, 
&  Peterson,  2001,  Fong,  Thorpe,  &  Baur,  2003;  Van  Erp  &  Padmos,  2003),  attentional  control 
(Chen  &  Joyner,  2007),  and  the  span  of  control  of  multiple  systems  (Bames,  Cosenzo,  Mitchell, 
&  Chen,  2005)  are  critical  to  safely  and  successfully  operating  unmanned  technologies. 

To  determine  if  these  cognitive  attributes  relate  to  objective  perfonnance  data  from 
operating  unmanned  technologies  in  an  ISR  mission,  a  small  controlled  field  test  was  conducted. 
The  field  test  used  the  Anny  Cognitive  Readiness  Assessment  (ACRA)  system  and  one  PackBot 
SUGV  with  its  associated  Operator  Control  Unit  (OCU).  The  ACRA  system  is  an  Army-specific, 
field-useable  performance  test  battery  generation  system  developed  by  NIT,  Inc.,  under  a  Phase 
II  Small  Business  Research  Initiative  (SBIR)  grant.  It  consists  of  a  suite  of  12  cognitive  tests  that 
best  represent  16  critical  cognitive  attributes  of  the  warfighter.  The  battery  of  tests  and  their 
relationship  to  each  of  the  16  critical  cognitive  attributes  were  established  by  subject  matter 
experts  (SMEs)  and  input  from  cognitive  scientists  at  the  Anny  Research  Laboratory  (ARL)  and 
NTI,  Inc.  Four  of  the  twelve  cognitive  tests  in  ACRA  were  chosen  for  this  study:  (1)  Motion 
Inference  (time  estimation);  (2)  NovaScan  C  (multitasking,  spatial  manipulation,  working 
memory);  (3)  Rapid  Decision  Making  (reaction  time,  selective  attention);  and  (4)  Tower  of 
Hanoi  (executive  planning).  These  tests  were  selected  using  an  optimization  methodology  (i.e. 
T-Matrix).3  The  T-Matrix  optimization  selects  the  tests  most  related  to  a  set  of  cognitive 
demands  associated  with  tele-operating  robotic  assets  in  an  ISR  mission. 


3  See  O’Donnel,  Moise,  &  Schmidt,  2004  for  a  detailed  description  about  the  T-Matrix  optimization  methodology. 
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This  study  was  conducted  on  a  small  area  where  several  high  mobility  and  multipurpose 
wheeled  vehicles  (HMMWV)  and  other  motor  pool  inventory  were  parked  (hereafter  referred  to 
as  the  PackBot  Track).  Seven  Soldiers  drove  a  PackBot  (Figure  la)  non-line  of  sight,  around 
seven  HMMWVs  while  searching  for  four  targets.  Soldiers  drove  the  PackBot  using  an  OCU, 
such  as  the  one  shown  below  in  Figure  lb.  Each  Soldier  was  informed  that  their  performance 
would  be  detennined  by  how  well  they  were  able  to  complete  the  task  and  the  number  of  targets 
identified.  Overall  perfonnance  was  based  on  how  fast  they  completed  the  course;  the  number  of 
targets  identified;  and  the  number  of  errors  committed  (i.e.  hitting  a  parked  HMMWV  or  reckless 
driving). 

Figure  la  and  b:  PackBot  SUGV  and  Operator  Control  Unit  (OCU) 


Once  the  Soldiers  finished  their 
training  on  ACRA  and  the 
PackBot,  each  were  allowed  a  15 
minute  practice  session  to  drive  the 
SUGV  non-line  of  sight  on  the 
PackBot  Track.  This  practice 
session  gave  each  Soldier  an 
opportunity  to  become  familiar 
with  the  rules  of  the  test  and  the  course.  Prior  to  the  start  of  each  test  session,  Soldiers 
completed  the  ACRA  testing  on  a  laptop.  Upon  completion  of  testing,  the  Soldier  reported  to  the 
PackBot  Track  where  he  received  oral  instructions  for  completing  the  course.  Soldiers  were  not 
allowed  to  reverse  the  course,  and  were  timed  using  a  stopwatch.  As  each  Soldier  completed  the 
course,  two  ARL  researchers  followed  behind  the  PackBot  and  recorded  any  errors.  This  testing 
routine  was  repeated  twice  (2  trials)  over  a  two  day  period. 


Results 

Situational  Awareness 

The  data  from  the  SALUTE  -AP  were  aggregated  by  category  (i.e.  size,  location, 
uniform,  equipment,  assessment,  and  prediction)  and  are  displayed  in  Figure  2.  These  scores 
show  that  S  A  within  each  category  of  the  SALUTE -AP  varied  by  day. 
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Figure  2  SA  by  Category  by  Day 


These  data  were  analyzed  using  a  repeated  measure  multivariate  analysis  of  variance 
(MANOVA)  to  determine  if  Soldiers  differed  in  their  ability  to  gain  SA  in  each  category 
identified  on  the  SALTUE-AP  by  day.  The  results  show  a  main  effect  for  SA  data  with  regard  to 
three  of  the  seven  categories  for  reporting  OPFOR  activity;  size,  location,  and  prediction  (see 
Table  2  below).  The  category  of  “time”  was  not  included  in  the  analysis  as  this  variable  was 
treated  as  a  constant  across  each  run  to  help  control  the  overall  variability  of  the  SA  data  as  it 
related  to  administering  the  survey.  Problems  in  data  collection  regarding  the  “activity”  question 
required  that  these  data  be  excluded  from  the  analysis.  The  three  variables  allow  the  comparison 
of  Soldiers’  understanding  of  the  size  of  the  enemy  force,  their  location,  and  probable  future 
activity  as  these  varied  by  day  of  mission. 


Table  2  Significant  Results 


SALUTE-AP 

Category 

Assumption 

Significance 

Size 

Sphericity  Assumed 

F(4,16)  =  3.529 

.030 

Location 

Sphericity  Assumed 

F(4,16)  =  3.067 

.047 

Prediction 

Sphericity  Assumed 

F(4,16)  =  3.695 

.026 

Pairwise  comparisons  were  computed  to  determine  which  days  were  significantly 
different  for  each  SALUTE-AP  category.  The  results  of  this  test  are  displayed  in  Table  3.  This 
analysis  shows  that  Soldiers’  understanding  of  the  size  of  the  enemy  force  and  their  location  was 
lower  on  Day  1.  Soldiers’  predictions  of  future  activities  of  the  OPFOR  were  more  accurate  on 
Day  1  and  Day  2  of  testing.  This  may  be  largely  due  to  the  differences  in  location  on  both  of 
these  days  (e.g.  open  areas  that  made  it  easier  to  detect  activity). 
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Table  3  Pairwise  Comparisons 


Pair 

Mean  Difference 

Standard  Error 

Significance 

Size 

Day  1  -  Day  2 

-.286 

.079 

.022 

Day  1  -  Day  3 

-.680 

.120 

.005 

Location 

Day  1-  Day  2 

-.580 

.15 

.018 

Day  1  -  Day  3 

-.960 

.172 

.005 

Prediction 

Day  1  -  Day  3 

-.800 

.228 

.025 

The  following  results  of  the  SALUTE -AP  survey  compares  average  levels  of  SA  by  day 
and  by  location.  An  average  score  was  created  for  the  platoon  by  combining  individual  scores 
for  the  level  1,  2,  and  3  questions.  These  scores  are  displayed  in  Figure  4  and  show  that  for  each 
record  run,  SA  scores,  on  average  are  fairly  high. 


Figure  3  Average  Situation  Awareness  by  Day 


Most  scores  fall  within  the  1.5  to  3.0  range.  The  scores  ranged  from  low  (1)  to  high  (3). 
Many  of  the  Soldiers  demonstrated  a  high  level  of  SA  as  measured  by  the  questions  of  enemy 
activity.  These  high  SA  scores  are  due,  in  part,  to  the  integrated  battle  command  system  that 
allowed  Soldiers  to  push  and  pull  information  across  the  network.  This  enhanced  FBCB2  system 
allowed  the  platoon  leaders  (PL  and  RNCO)  to  communicate  and  obtain  sensor  images  from 
distributed  SUGVs,  a  family  of  UGS,  and  the  higher  echelon  UAS  when  it  was  available.  The 
higher  SA  scores  on  20  and  25  July  are  due  in  part  to  the  availability  of  the  UAS.  The  UAS 
imagery  provided  an  overall  picture  of  the  area  of  operations,  thus  allowing  Soldiers  to  make 
sense  of  aggregate  information  from  ground  sensors  (FUGS  and  SUGV).  The  difference  in  SA 
scores  by  day  is  also  due  in  part  to  the  different  locations  where  ISR  missions  were  conducted. 
As  previously  stated,  ISR  missions  were  conducted  in  three  separate  locations.  This  allowed 
Soldiers  and  the  multidisciplinary  research  team  an  opportunity  to  use  the  full  capabilities  of  the 
range  facilities  at  Ft.  Dix,  NJ  and  to  study  the  behavior  of  each  system  in  different  conditions. 
The  location  of  ISR  missions  differed  by  day  due  to  range  availability. 
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Figure  4  demonstrates  how  average  SA  scores  differed  by  location.  In  the  9D  and  12A 
sites,  mean  scores  for  levels  1,  2,  and  3  were  similar.  In  the  MOUT  site,  scores  for  level  2  and  3 
(assessment  and  prediction)  were  noticeably  lower.  Unlike  the  9D  and  12A  sites,  researchers 
could  only  access  the  MOUT  site  on  one  day  due  to  limited  availability.  On  the  day  that  the 
mission  was  run  in  the  MOUT  site,  many  technical  problems  were  encountered  with  the 
unmanned  assets  and  the  communications  systems,  resulting  in  much  confusion  among 
participants.  However,  level  1  scores  were  relatively  high  for  this  mission,  as  they  were  on  the 
other  days.  As  Figure  4  shows,  most  scores  were  above  2,  suggesting  better  than  average  SA  for 
levels  1,  2,  and  3  for  TAC  9D  and  12A,  and  level  1  for  the  MOUT  site.  The  higher  scores  for 
level  1  indicate  that  the  unmanned  assets  allowed  the  Soldiers  to  accurately  report  the  number, 
location,  and  equipment  of  the  enemy  force.  The  lack  of  the  UAS  in  support  of  this  mission,  and 
the  frequent  problems  with  voice  communication  contributed  to  the  inability  to  develop  higher 
levels  of  understanding  and  projection.  This  is  a  key  finding  and  suggests  that  unmanned  images 
alone  cannot  substitute  for  human  communications  through  a  stable  network. 


Figure  4  Average  Situation  Awareness  by  Location 


To  demonstrate  how  SA  scores  varied  by  position,  a  representative  sample  of  one  day’s 
SA  scores  is  presented  in  Figure  5.  These  scores  show  how  scores  differed  by  position  for  level 
1,  2,  and  3  SA.  The  positions  reported  in  the  figures  represent  the  operators  of  the  unmanned 
systems  and  the  Robotics  NCO  (RNCO).  The  operators  of  the  unmanned  systems  had  views  of  a 
portion  of  the  battlespace,  depending  upon  where  their  systems  were  deployed  or  emplaced.  The 
FBCB2  operator  and  the  Robotics  NCO  were  both  integrating  information  from  the  sensor 
systems  on  the  command  interface.  Thus,  they  had  a  better  overall  perspective  of  the  enemy 
actions,  but  this  understanding  required  more  time  to  process.  The  Robotics  NCO  also 
represented  the  highest  enlisted  ranking  individual  in  this  group;  as  such  he  was  better  able  to 
place  his  understanding  in  perspective  and  anticipate  possible  enemy  actions  in  the  future.  The 
scores  represented  in  Figure  5  reflect  one  operator  per  position,  thus  the  box  plot  figure  is 
displayed  as  a  line  rather  than  a  box  reflecting  many  scores. 
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Figure  5  SA  Levels  1  -  3  by  Position,  20  July 


level  1  SA  level  2  SA  level  3  SA 


Typically,  the  assessment  (level  2)  and  prediction  (level  3)  of  activities  are  more  difficult  than 
simply  reporting  known  activity  (level  1).  Given  that  levels  2  and  3  require  greater  cognitive 
processes,  we  expect  that  these  levels  would  be  more  difficult  to  achieve  than  level  1  (Endsley, 
2000;  Matthews  &  Beal,  2002).  However,  on  20  July  observers  noted  an  exceptionally  well- 
connected  network  and  a  change  in  how  the  Platoon  used  their  unmanned  assets.  During  the 
after  action  review  on  the  previous  day,  Soldiers  reported  that  they  experienced  significant 
problems  with  maintaining  connectivity  to  both  SUGVs.  When  the  SUGVs  were  working,  they 
were  collocated  and  not  spread  across  the  area  of  operations  (AO).  This  arrangement  was 
determined  by  the  Robotics  Non-Commissioned  Officer  (RNCO)  and  the  Platoon  Leader  (PL)  as 
not  being  the  most  optimal  use  of  the  SUGVs.  Also  on  this  day,  the  FBCB2  systems  in  various 
vehicles  were  not  functional.  These  issues,  taken  together,  handicapped  the  Platoon’s  ability  to 
observe  OPFOR  activities  within  the  AO.  Thus  in  preparation  for  the  second  run  of  ISR 
missions  in  TAC  12A,  the  Soldiers  re-positioned  themselves  and  the  SUGVs  to  maximize 
network  connectivity.  The  PL  and  RBNCO  separated  the  SUGVs  so  that  each  viewed  a  separate 
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area  within  the  AO.  These  changes  resulted  in  a  better  use  of  the  network  to  share  information 
and  resulted  in  a  significant  gain  in  SA  (see  Figure  6). 

Figure  6  Average  Situational  Awareness  by  Location  by  Day 


In  Case  1  -  TAC  9D,  SA  was  not  significantly  different  between  runs.  However,  higher 
SA  on  the  second  run  in  TAC  9D  should  not  be  surprising  considering  Soldiers  were  drawing 
from  an  accumulation  of  lessons  learned  and  experiences  since  the  start  of  the  exercise.  Soldiers 
reported  better  SA  on  the  second  run  in  TAC  12A;  F  (1,7)  =  36.59,  p  =  .001.  As  previously 
stated,  observations  from  the  data  collection  team  suggest  that  the  increase  in  SA  may  be  a  result 
of  Soldiers  reporting  less  problems  with  achieving  continued  connectivity  and  that  they  used 
each  system  more  efficiently  and  expertly  to  obtain  information.  Overall,  SA  is  lower  for  the 
MOUT  site  when  compared  to  ISR  missions  in  TAC  12A  and  the  TAC  9D.  This  effect  may  be 
best  explained  by  the  general  complexity  for  conducting  coordinated  ISR  missions  in  urban 
terrain.  Abandoned  buildings,  cars  and  random  debris  provide  obstacles  that  interfere  with  the 
maneuverability  of  these  unmanned  systems  and  also  threaten  continuous  connectivity  for 
information  transfer.  In  conclusion,  SA  was  moderate  in  most  locations.  Compared  to  2006  data 
collection  where  a  ceiling  effect  was  suspected  for  SA  analysis,  these  data  appear  to  describe  a 
more  realistic  pattern  of  SA.  This  was  aided  by  researchers’  attempts  to  complicate  the  scenarios 
by  varying  OPFOR  activity  and  using  various  types  of  terrain  features  for  ISR  missions. 

The  MARS  survey  provided  another  key  piece  of  information;  subjective  SA.  The 
questions  were  measured  on  a  4  point  scale  (l=very  easy,  4=very  difficult)  and  were 
administered  at  the  conclusion  of  the  mission.  The  four  questions  ask  Soldiers  to  rate  the 
difficulty  they  experienced  in  identifying  mission-critical  cues,  to  understand  activity,  to  predict 
what  was  going  to  occur,  and  how  to  best  achieve  goals  in  the  mission.  The  data  displayed  in 
Figure  7  demonstrates  that  the  majority  of  Soldiers  (90%  (N=10))  agreed  that  it  was  easy  or  very 
easy  to  identify  mission  critical  cues  using  surrogate  FCS  unmanned  systems  in  open  terrain 
(DTA-5)  compared  to  80%  in  forested  terrain  (TAC  12A)  and  73%  in  the  MOUT  Site.  These 
high  scores  are  likely  due  to  the  capabilities  of  the  technologies  and  emergent  TTPs  for  using 
each  technology  in  the  different  field  enviromnents.  Consistent  with  our  expectations,  the 
characteristics  of  the  open  area  (TAC  9D)  was  much  easier  to  detect  and  identify  cues  during 
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ISR  mission  because  this  area  lacked  trees  or  buildings  which  may  have  provided  more 
obstructions  to  viewing  OPFOR  activity. 

Figure  7  Average  Subjective  SA 

Ability  to  Detect  and  Understand  Cues  During  ISR 
Missions  by  Location 

Percent  of  Soldiers  □  Easy'  to  'Very  Easy' 


When  comparing  objective  and  subjective  SA,  we  see  a  positive  trend.  Soldiers 
perceived  that  they  were  able  to  achieve  high  levels  of  SA  with  very  little  effort,  and  their 
objective  scores  for  SA  are  consistently  high. 


Workload 

The  NASA  Task  Load  Index  was  used  to  measure  perceptions  of  workload.  This 
measure  requires  participants  to  rate  their  workload  according  to  six  dimensions.  These  are 
mental  and  physical  workload,  temporal  demand  (e.g.  time  pressure  felt),  satisfaction  with  own 
performance,  effort,  and  frustration  felt.  These  questions  are  measured  on  a  10  point  scale. 
Participants  completed  this  survey  at  the  conclusion  of  each  day. 

Multiple  Analysis  of  Variance  (MANOVA)  and  ANOVAs  were  conducted  for  the 
workload  data.  These  analyses  investigated  the  differences  in  workload  reported  by  the 
individual  Soldiers. 

The  MANOVA  showed  a  main  effect  for  Components  of  Workload,  F  (5,  30)  =  20.47,  p 
<  .00  and  a  main  effect  of  Day,  F  (4,24)  =  4.69,  p  <  .00.  To  determine  which  component  of 
workload  contributed  to  the  main  effects,  pairwise  comparisons  were  run.  These  data  show  that 
mental  workload  was  the  highest  component  (M  =  63.57  (5.87)),  followed  by  temporal  demand 
(M  =  63.93  (7.13))  and  frustration  (M  =  65.71  (6.35)).  Visual  inspection  of  the  descriptive 
statistics  for  mean  reports  of  workload  by  day  show  that  the  highest  average  workload  was 
reported  on  day  two  (M  =  62.74  (5.05)).  Comparisons  of  these  data  with  average  SA  reported 
earlier  in  Figure  6  indicate  that  the  higher  workload  ratings  on  day  two  may  be  a  potential  cause 
of  the  lower  SA  scores  for  that  day. 
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Army  Cognitive  Test  Battery 


Soldiers  maneuvered  a  PackBot  SUGV  through  a  carefully  designed  obstacle  course 
where  they  were  required  to  identify  targets,  maneuver  through  a  course  while  executing  several 
functions  of  the  vehicle.  This  was  a  timed  course  with  a  data  collector  following  the  Soldier  to 
record  target  detections,  errors,  or  unfinished  course  elements. 

Accuracy  and  efficiency  scores  for  each  Soldier  are  shown  in  Figure  11.  Low 
performance  scores  indicate  higher  efficiency  (less  time  to  complete  course,  more  targets 
identified,  and  little  errors).  Overall,  Soldiers  improved  their  performance  between  trials.  This 
was  expected  as  Soldiers  committed  fewer  errors  and  identified  more  targets  in  the  second  trial. 

Figure  8  Adjusted  Completion  Times  on  PackBot  Track 
Adjusted  Time  to  Complete  PACKBOT  Track 

_ ■  Trial  1 

■  Trial  2 

fb  n  .  n  ■  m 


I  I  IB  I  II  I 


si  s2  s3  s4  s5  s6  s7 
Operator 

Note.  Adjusted  time  (AT)  is  computed  as  a  function  of  the  total  time  to  complete  course  (TT),  the  number  of 
targets  identified  (TI),  the  number  of  targets  missed  (TM),  and  the  number  of  errors  (E). 

AT  =  [  TT  +  (E  x  30s)  +  (TM  x  60s)  ]  -  (TI  x  60s). 

Non-parametric  correlations  were  computed  to  assess  the  relationships  between  ACRA 
subtests  and  the  PackBot  Track  performance  measures.  It  was  hypothesized  that  the  ACRA  tool 
would  demonstrate  significant  relationships  with  the  PackBot  Track  performance  metrics 
relevant  to  common  cognitive  skill-sets  for  operating  unmanned  technologies.  Scores  on  ACRA 
and  the  PackBot  Track  were  averaged  across  trials.  Due  to  the  large  number  of  ACRA  metrics, 
only  a  small  number  were  selected  for  analysis.  Only  the  variables  for  which  strong  correlations 
exist  are  provided  in  Table  4. 
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Table  4  ACRA  and  PackBot  Track  Correlations 


Time  to  Complete 
Course  (sec) 

Targets 

Identified 

Errors 

Rapid 

Decision 

Ave.RT 

P 

-.879 

.87 

Making 

Sig 

.009 

.011 

Time 

Ave.  Error 

Estimation 

P 

-.875 

Sig 

.010 

Spatial 

Orientation 

Per.  Cor 

P 

.735 

Sig 

.060 

Three  of  four  ACRA  subtests  were  strongly  related  to  at  least  one  performance  measure 
from  the  PackBot  Track.  Rapid  decision  making  demonstrated  relation  to  the  number  of  targets 
identified  and  the  number  of  errors  committed.  Time  estimation  (as  measured  by  the  Motion 
Inference  Test)  also  demonstrated  a  relationship  to  the  PackBot  Track  metrics.  The  overall  time 
taken  to  complete  the  PackBot  Track  is  significantly  correlated  with  an  ability  to  infer  motion 
and  the  perception  of  motion.  Finally,  spatial  orientation  (as  measured  by  the  Manikin  Test) 
percent  correct  revealed  relation  to  the  overall  time  taken  to  complete  the  PackBot  Track. 

Results  on  the  Tower  of  Hanoi  subtest  revealed  little  relationship  to  any  of  the  PackBot 
Track  perfonnance  measures.  This  could  be  an  artifact  of  the  perfonnance  test,  which  required 
little  planning  and  problem  solving  (as  measured  by  the  Tower  of  Hanoi). 

The  above  results  demonstrate  the  overall  validity  and  sensitivity  of  the  ACRA  to 
unmanned  system  operations.  The  results  also  highlight  how  specific  cognitive  metrics  are 
related  to  objective  performance.  However,  given  the  small  sample  size  and  the  larger  number 
of  ACRA  metrics,  additional  research  must  be  conducted.  It  is  strongly  recommended  that 
additional  research  include  a  variety  of  controlled  user  testing  involving  different  operational 
tasks  for  tele-operating  unmanned  systems. 


Conclusions 

The  C4ISR  technologies  used  in  this  activity,  when  used  independently,  provide  robust 
and  important  contributions  to  Intelligence,  Surveillance  &  Reconnaissance  (ISR)  tasks  within  a 
platoon  organization.  When  data  from  these  technologies  are  combined  in  a  suite  of  tools,  it 
becomes  a  challenge  to  display  the  information  in  such  a  way  that  will  not  overwhelm  the 
Soldiers’  abilities  to  develop  actionable  knowledge.  Effective  networked  operations  require  the 
synchronization  of  the  physical  and  the  cognitive/social  levels  of  the  architecture.  In  addition  to 
evaluating  a  network’s  capability  to  transmit  data  within  a  time  interval  and  among  nodes,  we 
must  concentrate  on  delivering  the  minimal  amount  of  data  that  has  the  highest  impact  on  Soldier 
decision  making.  The  methods  used  in  this  activity  were  useful  in  documenting  the  impact  of  the 
network  on  Soldier  SA  and  identifying  capability  gaps. 
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The  cognitive  attributes  of  rapid  decision  making,  time  estimation,  and  spatial  orientation 
were  demonstrated  to  be  significantly  correlated  with  high  performance  on  PackBot  SUGV  tasks 
of  target  detection,  vehicle  maneuverability,  and  course  completion  times.  This  finding,  when 
combined  with  observations  and  reports  of  problems  associated  with  Soldiers’  use  of  the  SUGV 
Operator  Control  Unit  (OCU),  can  be  used  to  drive  enhancements  to  OCU  features. 

The  use  of  the  SALUTE-AP  situational  awareness  survey  demonstrated  that  this 
innovative  measure  was  robust  in  a  field  test  and  useful  for  evaluating  individual  Soldier  SA  at 
three  levels.  Though  mean  levels  of  SA  were  high  in  all  test  environments,  the  MOUT  site  did 
introduce  more  uncertainty  and  less  awareness  of  OPFOR  activity.  This  is  undoubtedly  a  natural 
conclusion  and  suggests  that  additional  support  is  needed  in  the  decision  making  process. 
Findings  that  the  Robotics  NCO  had  the  highest  levels  of  reported  workload  are  consistent  with 
the  requirement  for  infonnation  fusion  from  the  unmanned  systems  prior  to  sending  reports  to 
the  higher  headquarters. 

The  results  from  the  ACRA  tests  demonstrate  the  overall  validity  and  sensitivity  of  the 
test  battery  to  unmanned  system  operations.  The  results  also  highlight  how  specific  cognitive 
metrics  are  related  to  objective  performance.  However,  given  the  small  sample  size  and  the 
larger  number  of  ACRA  metrics,  additional  research  must  be  conducted.  It  is  strongly 
recommended  that  additional  research  include  a  variety  of  controlled  user  testing  involving 
different  operational  tasks  for  tele-operating  unmanned  systems. 

This  experiment  demonstrates  how  the  introduction  of  unmanned  technologies  can  assist 
in  the  rapid  development  of  situational  awareness  among  a  reconnaissance  platoon.  However, 
this  experiment  also  demonstrated  that  these  technologies  are  used  at  a  cost  to  the  human 
operators.  These  costs  include  physical  workload,  mental  workload,  and  network  connectivity. 
The  unmanned  systems  require  human  intervention  to  emplace  or  operate  the  sensors  and  replace 
batteries  or  extract  the  vehicles  from  untenable  situations  (e.g.  holes,  ruts  or  obstructed  sight). 
The  operators  and  decision  makers  must  expend  cognitive  effort  to  accurately  geo-locate  images 
detected  by  sensors;  this  is  more  difficult  than  most  would  expect.  The  annotation  image 
provided  by  the  ARL  Packbot  OCU  was  an  excellent  example  of  making  this  process  much 
easier.  Finally,  the  highest  levels  of  SA  were  observed  only  when  the  voice  communications 
were  working  with  the  unmanned  sensor  images.  The  sensor  images  alone  were  not  sufficient  to 
portray  the  enemy  picture  to  the  decision  makers.  Network  performance  that  allows  the 
transmission  of  voice  or  text  communication  in  addition  to  sensor  images  is  essential  to  rapid 
decision  making. 
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Plan  of  Discussion 

•  Introduction  to  experiment  setting 

•  The  distributed  communications  network 

•  Unmanned  sensor  systems 

•  Experiment  design 

•  Dependent  measures 

•  Results 

•  Conclusions 


C4ISR  On  The  Move  Campaign  of  Experimentation  Charter: 

Provide  a  relevant  environment/venue  to  assess  emerging 
technologies  in  a  C4ISR  System-of-Systems  (SoS) 
configuration  to  enable  a  Network  Centric  Environment  IOT 
mitigate  risk  for  FCS  Concepts,  Future  Force  technologies  and 

accelerate  technology  insertion  into  the  Current  Force. 

•  Perform  Systems  of  Systems  (SoS)  integration 

•  Objective  hardware  and  software 

•  Surrogate  &  Simulated  systems  as 
necessary  due  to  maturity,  availability  and 
scalability 

•  Conduct  Technical  Live,  Virtual,  and 
Constructive  technology  demonstrations 

•  Component  Systems  Evaluations 

•  Scripted  end-to-end  SoS  Operational 
Threads 

•  Technology  experiment/assessment  in  a 
relevant  environment  employing  Soldiers 

•  Develop  test  methodologies,  assessment  metrics, 
automated  data  collection  and  reduction,  and 
analysis  techniques 


Bv  authority  of  the  Army  Acquisition  Executive 
and  by  the  designation  of  the  undersigned 
as  the  Commander,  RDECOM, 

I  hereby  charter 

LTC  William  T.  Utroska 

as  the  Product  Manager  for 

C4ISR  ON-TH  E-MOVE 

in  accordance  with  the  Army  Acquisition  Management  Process. 

As  Product  Manager  for  this  program,  you  will  perform  as  the  Army  centralized 
manager  reporting  to  the  Commander,  RDECOM. 

You  will,  as  the  responsible  management  official,  manage  your  assigned 
program  in  a  manner  consistent  with  the  policies  and  principles  articulated 
in  Department  of  Defense  directives  and  Acquisition  Reform  Initiatives.  In 
addition,  you  are  responsible,  along  with  your  strategic  partners,  Army  Materiel 
Command,  Defense  Contract  Management  Agency,  Defense  Logistics  Agency,  and 
Army  Test  and  Evaluation  Command,  for  the  Life  Cycle  Management  of  your 
program  and  will  actively  manage,  lu  the  best  of  your  abilities,  within  approved 
resources,  program  cost)  performance  and  schedule  and  provide  assessments  of 
program  status,  risk  and  contractor  performance. 

You  are  hereby  delegated  the  full-line  authority  of  the  Commander,  RDECOM 
for  the  centralized  management  of  your  assigned  program.  You  will 
be  provided  with  timely  decisions  by  Senior  Leadership  and  are  expected  to  be 
candid  and  forthcoming  without  fear  of  personal  consequences. 


This  charier  remains  in  effect  until  terminated  or  superseded. 


and  Engineering  Command 


SENSORS 


Ft.  Dix  Range 


Fleet  of  instrumented 
vehicles 


Daily  Mission  Runs 


Test  Range 

Characteristics 

Dates 

Case  1, 

TAC  9D 

Open  rolling  sandy 
areas  with  lightly 
forested  sections. 

18  and  25 
July  2007 

Case  2, 

TAC  12A 

Open  sandy  areas  with 
heavily  forested 

sections  including 

‘Vietnam  Village’,  a 
set  of  three  huts  in  a 
forested  area. 

19  and  20 
July  2007 

Case  3, 
MOUT 

An  urban  site 
configured  in  a 
triangular  shape. 

24  July 

2007 
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Participants 

•  10  Soldiers  from  NJ  ARNG 

•  Soldiers  were  organized  into  elements  of  a  reconnaissance 
platoon 

•  Key  leader  positions  included  Platoon  Leader,  Platoon 
Sergeant,  Robotics  NCO,  and  two  dismounted  recon  squads 
with  sensors 

•  Sensors  included: 

-  Class  1  Unmanned  Air  System  (UAS) 

-  PackBot  Unmanned  Ground  Vehicle  (UGV) 

-  Family  of  Unattended  Ground  Sensors  (UGS) 

•  Soldiers  operated  ISR  scenarios  against 
an  Opposing  Force  (OPFOR) 

•  Scenarios  were  designed  to  replicate  current  threats  (IED 
emplacement,  weapons  storage,  prisoner  exchange,  high 
value  target  meeting) 


Procedure 


Training 

-  Equipment 

-  FBCB2 

Scripted  scenarios  conducted  over  5  days 
Dependent  Measures 

-  Situational  Awareness 

-  Workload:  Mission  Awareness  Rating  Scale 
(Matthews  &  Beal,  2002)  and  NASA  TLX  (Hart  & 
Staveland,  1988) 

Cognitive  Predictors  of  UGV  Performance: 
Excursion  Experiment  to  validate  cognitive  test 
battery 


Data  Collection  Methods 
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Instrumentation,  Field/Lab 
Data  Collection, 
Interviews,  Observation, 
Reduction,  Analysis 


SALUTE-AP 


OPFOR  Action 
(Ground  Truth) 

Size 
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Location 

Uniform 
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Activity  is 
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5 

Group  may 
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5 

•  Based  on  SAGAT  (Endsley,  2000) 

•  Utilizes  standard  Army  reporting  categories 

•  Causes  minimal  intrusion  ICCRTS  Presentations: 

•  Easy  to  train  Bowman  &  Kirin,  2006 

•  Easy  to  administer  and  analyze  Bowman  &  Thomas,  2007 

•  Scored  on  low-medium-high  scale 


SA:  Mission  Awareness  Rating  Scale 

Answered  on  a  4  point  scale  from  low  to  high: 

1.  Please  rate  your  ability  to  identify  mission- critical  cues  in  this  mission. 

2.  How  well  did  you  understand  what  was  going  on  during  the  mission? 

3.  How  well  could  you  predict  what  was  about  to  occur  next  in  the  mission? 

4.  How  aware  were  you  of  how  to  best  achieve  your  goals  during  this  mission? 

SOURCE: 

Matthews,  M.  &  Beal,  S.  (2002).  Assessing  situation  awareness  in  field  training  exercises.  Alexandria,  VA:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences. 

McGuinness,  B.,  &  Ebbage,  L.  (2002).  Assessing  human  factors  in  command  and  control:  workload  and  situational  awareness 
measures.  Proceedings  of  the  2002  Command  and  Control  Research  and  Technology  Symposium,  Monterey,  CA. 

Workload:  NASA  TLX 

1.  Based  on  six  scales  (mental  demand,  physical  demand,  temporal  demand, 
performance,  effort,  and  frustration  level. 

2.  Ratings  on  6  scales  are  weighted  according  to  subject’s  evaluation  of 
relative  importance. 

3.  Ratings  range  from  1-100. 

SOURCE:  Hart  &  Staveiand,  1988 


SA  Results:  Report  type 

•  On  average,  subjects  differed  in  their  ability  to  identify  size, 
location,  and  possible  future  activities. 

•  Size,  Location,  and  Prediction  were  statistically  significant 
variables 
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SA  Results:  Effect  of  Day 

•  Level  1  and  Level  2  and  3  SA  variable 

•  20  and  25  July  high  scores  influenced  by  availability 
of  Class  1  UAS 


SA  Results:  Effect  of  Location 

SA  results  for  Level  1,  2,  and  3  were  consistent  in  open 
rolling  terrain  with  slight  vegetation. 

SA  results  for  level  2  and  3  dropped  noticeably  in  MOUT 
site  where  OPFOR  activity  was  more  complicated  and 
more  difficult  to  understand  and  predict. 


SA  Results:  Effect  of  Role 

Slight  differences  in  level  1,  2,  and  3  for  all  operators;  most  noticeably 
UGS  operator  in  level  3. 

With  respect  to  level  1,  UGS  and  UGV  operators  had  higher  levels  of  SA 
due  to  robotics-eyes-on  activity. 
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MARS  SA  Results 


Consistent  results  to  SALUTE-AP  showing  that  SA  was 
easier  to  achieve  in  open  areas  compared  to  MOUT  site. 


Village 

Location 


Workload 


•  Mental  Workload  was  the  highest 
component  of  workload  (M=63.57  (5.87), 
followed  by  temporal  demand  (M  =  63.93 
(7.13))  and  frustration  (M  =  65.71  (6.35)). 

•  Highest  average  workload  was  reported  on 
day  two  (M  =  62.74  (5.05)). 

•  High  workload  on  day  two  could  be  an 
explanation  for  lower  SA  scores  on  that  day. 

•  Frustration  was  related  to  technology 
problems. 


Army  Cognitive  Readiness 
Assessment  Test  Battery 

*  Designed  to  predict  performance  with  the  Packbot 
Unmanned  Ground  Vehicle  (UGV)  system  with  9 
Soldiers  in  two  obstacle  course  settings. 

*  Rapid  decision  making,  time  estimation,  and  spatial 
orientation  were  found  to  be  highly  correlated  to  SUGV 
operation. 


I  111  1 1 1 1 


Conclusions 


A  suite  of  ISR  technologies  prove 
challenging  to  users 

-  Display  and  information  overload 

-  Difficult  to  synchronize  information  from 
various  sensors 

Cognitive  attributes  of  rapid  decision 
making,  time  estimation,  and  spatial 
orientation  were  demonstrated  to  be 
correlated  with  operating  a  UGV 


Conclusions 


SALUTE-AP  method  robust  to  field  conditions 
and  a  valid  measure  of  SA  levels 

Suite  of  sensors  contributed  to  high  level  of  SA 
in  short  time  period,  but  at  costs  of  physical  and 
mental  workload 

The  ability  to  predict  performance  in  operating 
these  systems  can  optimize  training  time  and 
reduce  expensive  operational  accidents  from 
unskilled  operators. 


